Object matching for tracking, indexing, and search

ABSTRACT

A camera system comprises an image capturing device, object detection module, object tracking module, and match classifier. The object detection module receives image data and detects objects appearing in one or more of the images. The object tracking module temporally associates instances of a first object detected in a first group of the images. The first object has a first signature representing features of the first object. The match classifier matches object instances by analyzing data derived from the first signature of the first object and a second signature of a second object detected in a second image. The second signature represents features of the second object derived from the second image. The match classifier determine whether the second signature matches the first signature. A training process automatically configures the match classifier using a set of possible object features.

RELATED APPLICATIONS

This application claims benefit under 35 U.S.C. §119(e) of U.S.Provisional Application No. 61/033,349, entitled “Method of DynamicObject and Event Classification,” filed Mar. 3, 2008, and U.S.Provisional Application No. 61/033,284, entitled “Method and System forTracking Objects Under Video Surveillance,” filed Mar. 3, 2008, both ofwhich are incorporated herein by reference in their entireties.

TECHNICAL FIELD

This disclosure relates generally but not exclusively to videosurveillance and more particularly to matching objects represented inmultiple images.

BACKGROUND INFORMATION

Automated security and surveillance systems typically employ videocameras or other image capturing devices or sensors to collect imagedata. In the simplest systems, images represented by the image data aredisplayed for contemporaneous screening by security personnel and/orrecorded for later reference after a security breach. In those systems,the task of detecting objects of interest is performed by a humanobserver. A significant advance occurs when the system itself is able toperform object detection and tracking, either partly or completely.

In a typical surveillance system, for example, one may be interested intracking a detected object such as a human, a vehicle, an animal, etc.that moves through the environment. Existing systems capable of trackingdetected objects attempt to track objects using motion prediction andtracking of selected features in consecutive frames of video. Othertechniques, such as the SIFT method, attempt to precisely represent theappearance of an instance of an object such that the representation canbe used to match multiple instances of an object irrespective of theirtemporal proximity. Known tracking systems, however, suffer from one ormore of (1) inability to be trained, (2) lack of integration with objectsearching, indexing, and classification systems, (3) inadequate objecttracking and search performance, and (4) ineffective cross cameratracking capabilities.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a pictorial diagram of a camera system according to oneembodiment.

FIG. 2 is a simplified block diagram of one of the image capturingdevices in the system shown in FIG. 1.

FIG. 3 is a block diagram depicting of a method of generating an inputfor the match classifier of FIG. 2, according to one embodiment.

FIG. 4 is a block diagram of the match classifier of FIG. 2.

FIG. 5A is a block diagram showing one of the initial N−1 steps in thematch classifier shown in FIG. 4.

FIG. 5B is a depiction of acceptance and rejection thresholds utilizedin the step shown in FIG. 5A.

FIG. 6 is a flow chart depicting a method of utilizing the matchclassifier, according to one embodiment.

FIG. 7 is a flow diagram of a match classifier training method.

FIG. 8 is a flow chart of a match classifier training method.

FIG. 9 is a flow chart of another match classifier training method.

FIG. 10 is a block diagram of an object tracking system.

FIG. 11 is a flow chart of an object tracking method.

FIG. 12 is a hybrid block diagram and flow chart of another objecttracking system.

FIGS. 13A and 13B are hybrid block diagrams and flow charts of aninter-camera tracking method.

FIG. 14 is a block diagram of an object indexing system.

FIG. 15 is a pictorial diagram of an object search method.

DETAILED DESCRIPTION OF EMBODIMENTS Overview

With reference to the above-listed drawings, this section describesparticular embodiments and their detailed construction and operation.The embodiments described herein are set forth by way of illustrationonly and not limitation. Those skilled in the art will recognize inlight of the teachings herein that there is a range of equivalents tothe example embodiments described herein. Most notably, otherembodiments are possible, variations can be made to the embodimentsdescribed herein, and there may be equivalents to the components, parts,or steps that make up the described embodiments.

As one skilled in the art will appreciate in light of this disclosure,certain embodiments are capable of achieving certain advantages over theknown prior art, including some or all of the following: (1) unifyingobject tracking, indexing, and searching methods, (2) implementing atrainable match classifier for object tracking, indexing, and searching,(3) ability to search for instances of an object captured prior to arule violation, (4) ability to recognize previously captured objectsafter a new object type is added to a detection library, (5) ability toadapt object tracking, indexing, and searching to environmental changes,(6) ability to add new features to optimize detection and classificationaccuracy, (7) implementing a high-speed tracking system that allowsaccurate tracking even with relatively low frame rates, (8) implementingan accurate tracking system on an embedded platform using inexpensivehardware, (9) ability to improve object detection accuracy andclassification accuracy by feeding back accurate tracking data, and (10)ability to automatically track objects across cameras in real time.These and other advantages of various embodiments will be apparent uponreading the remainder of this section.

For the sake of clarity and conciseness, certain aspects of componentsor steps of certain embodiments are presented without undue detail wheresuch detail would be apparent to those skilled in the art in light ofthe teachings herein and/or where such detail would obfuscate anunderstanding of more pertinent aspects of the embodiments.

A camera system comprises an image capturing device, an object detectionmodule connected to the image capturing device, an object trackingmodule connected to the object detection module, and a match classifierconnected to the object tracking module. The image capturing device hasa field of view and produces image data representing multiple images ofthe field of view. The object detection module receives the image dataand is operable to detect objects appearing in one or more of themultiple images. The object tracking module is operable to temporallyassociate instances of a first object detected in a first group of themultiple images. The first object has a first signature representingfeatures of the first object derived from the images of the first group.The match classifier is operable to match object instances by analyzingdata derived from the first signature of the first object and a secondsignature of a second object. The second object is detected in a secondimage distinct from the images of the first group. The second signaturerepresents features of the second object derived from the second image.The match classifier analyzes the first set of data to determine whetherthe second signature matches the first signature. The match classifieroperates to match object instances in a manner that has been determinedat least in part by a training process that automatically configures thematch classifier using a set of possible object features.

A method tracks an object captured by a camera system. The methodcaptures first and second images of a field of view of the camerasystem, detects a first object captured in the first image and a secondobject captured in the second image, and produces a first signature ofthe first object and a second signature of the second object. The firstsignature represents a first set of features of the first object and thesecond signature represents a second set of features of the secondobject. The method produces data derived from the first and secondsignatures. The data corresponds to a first subset of features selectedfrom the first set and a second subset of features selected from thesecond set. A training process automatically determines the features ofthe first and second sets that are to be selected for inclusion in thefirst and second subsets. The method analyzes the data to determinewhether the second object is an instance of the first object.

A method generates index elements of objects in images captured by acamera system. The method captures a first group of multiple images ofan object in a field of view of the camera system, detects the object inthe first group of multiple images, and generates a first set ofsingle-frame signatures of the object. The single-frame signatures ofthe first set correspond to features of the object derived from themultiple images of the first group. The method produces a first averagesignature of the object. The first average signature is an average ofthe single-frame signatures of the first set. The first averagesignature is a first index element of the object. The method produces acumulative signature of the object. The cumulative signature is aweighted average of the single-frame signatures of the first set. Themethod captures a second image of the object, detects the object in thesecond image, and generates a second single-frame signature derived fromthe second image. The method updates the cumulative signature byweighing the second single-frame signature and averaging the weightedsecond single-frame signature with the cumulative signature. The methoddetermines whether to accept or reject the updated cumulative signatureand the first average signature as a match. The first average signatureis updated based on the second single-frame signature when the updatedcumulative signature and the first average signature are accepted as amatch. A second average signature of the object is produced based on thesecond single-frame signature when the updated cumulative signature andthe first average signature are rejected as a match. The second averagesignature is a second index element of the object.

A method searches data to identify images of an object captured by acamera system. The method captures a first object in an image of a fieldof view of the camera system. The first object has a first signaturerepresenting features of the first object. The method displays an imageof the first object on a display, accepts user input indicative of aselection of the first object, and makes determinations as to whetherthe first signature matches a group of signatures corresponding to a setof objects captured in images of the field of view of the camera system.The method sends a notification when a second signature of a secondobject of the set matches the first signature.

A camera system comprises an image capturing device, an object detectionmodule connected to the image capturing device, a user interfaceconnected to the image capturing device, a match classifier, an objecttracking module connected to the match classifier and the objectdetection module, and a search module connected to the match classifierand the user interface. The image capturing device has a field of viewand produces image data representing multiple images of the field ofview. The object detection module receives the image data and isoperable to detect objects appearing in one or more of the multipleimages. The user interface includes a display for displaying objectsdetected by the object detection module and an input device operable toselect objects detected by the object detection module in response touser commands. The match classifier is operable to match objectsdetected by the object detection module. The object tracking module isoperable to temporally associate instances of objects detected by theobject detection module and is also operable to call on the matchclassifier to determine whether to associate instances of objects. Thesearch module is operable to retrieve stored images of objects selectedby the input device and is also operable to call on the match classifierto locate a stored image of a selected object.

Camera System

FIG. 1 is a pictorial diagram of a camera system 100 according to oneembodiment. The camera system 100 includes image capturing devices 102,a user interface 104, and a remote storage/processing unit 106 (e.g., aremote server) connected to each other in a network 108. The network 108may include any type of wired or wireless network. Although the camerasystem 100 of FIG. 1 includes multiple image capturing devices 102connected in a network, the camera system 100 may include a single imagecapturing device 102. The image capturing devices 102 may include aninternal storage system 110, comprising a hard drive (HD) 111, and ametadata database (DB) 112. For example, the image capturing devices 102may include a storage system described in commonly owned U.S. patentapplication Ser. Nos. 12/105,971 and 12/105,893, entitled “Content AwareStorage of Video Data” and “Extending the Operational Lifetime of aHard-Disk Drive Used in Video Data Storage Applications,” respectively,both of which are incorporated herein by reference in their entireties.The user interface 104 includes a display 114 and an input device 116.Image capturing devices 102 capture images of their respective fields ofview and generate image data representing the images. It is to beunderstood that images may refer to still images or motion video images.The image data is communicated over the network 108 to the userinterface 104, and images of one more of the fields of view arepresented on the display 114. The input device 116 is operable to allowa user to provide user feedback information for the camera system 100.Image data may also be communicated over the network 108 to the remotestorage/processing unit 106.

FIG. 2 is a simplified block diagram of one of the image capturingdevices 102. The image capturing device 102 may be a high-resolutionvideo camera such as a megapixel video camera. The image capturingdevice 102 may also capture data from outside the visible spectrum(e.g., thermal energy). In addition to the storage system 110, the imagecapturing device 102 includes an image processing unit that includes avideo analytics module 200 for analyzing images captured by the imagecapturing device 102. The image processing unit need not be containedwithin a housing 202 of the image capturing device 102 as depicted inFIG. 2. Moreover, the remote storage/processing unit 106 may alsoinclude an image processing unit.

The video analytics module 200 includes a number of modules forperforming various tasks. For example, the video analytics module 200includes an object detection module 204 for detecting objects appearingin the field of view of the image capturing device 102. The objectdetection module 204 may employ any known object detection method suchas motion detection or blob detection. The object detection module 204may include the systems and use the detection methods described incommonly owned U.S. Pat. No. 7,627,171 entitled “Methods and Systems forDetecting Objects of Interest in Spatio-Temporal Signals,” the entirecontents of which are incorporated herein by reference.

The video analytics module 200 also includes an object tracking module206 connected to the object detection module 204. As used herein, theterm “connected” means logically or physically connected directly orindirectly through one or more intermediaries. The object trackingmodule 206 is operable to temporally associate instances of an objectdetected by the object detection module 204. The object tracking module206 generates metadata corresponding to objects it tracks. The metadatamay correspond to signatures of the object representing the objects'appearance or other features. The metadata may be transmitted to themetadata database 112 for storage. The object tracking module 206 isdescribed in more detail below.

The video analytics module 200 also includes a temporal objectclassification module 208. The temporal object classification module 208is operable to classify an object according to its type (e.g., human,vehicle, animal) by considering the object's appearance over time. Inother words, the object tracking module 206 tracks an object formultiple frames (i.e., multiple images), and the temporal objectclassification module 208 determines the object's type based upon itsappearance in the multiple frames. The temporal object classificationmodule 208 may combine information regarding the trajectory of an object(e.g., whether the trajectory is smooth or chaotic, whether the objectis moving or motionless) and the confidence of classifications made byan object classification module 210 averaged over multiple frames. Forexample, classification confidence values determined by the objectclassification module 210 may be adjusted based on the smoothness oftrajectory of the object. The temporal object classification module 208may assign an object to an unknown class until the object is classifiedby the object classification module a sufficient number of times and apredetermined number of statistics have been gathered. In classifying anobject, the temporal object classification module 208 may also take intoaccount how long the object has been in the field of view. The temporalobject classification module 208 may make a final determination aboutthe class of an object based on the information described above. Thetemporal object classification module 208 may also use a hysteresisapproach for changing the class of an object. For example, a thresholdmay be set for transitioning the classification of an object fromunknown to a human, and that threshold may be larger than a thresholdfor the opposite transition (i.e., from a human to unknown). Thetemporal object classification module 208 may generate metadata relatedto the class of an object, and the metadata may be stored in themetadata database 112.

The video analytics module 200 also includes the object classificationmodule 210 connected to the object detection module 204. In contrast tothe temporal object classification module 208, the object classificationmodule 210 determines an object's type based upon a single instance(i.e., single image) of the object. The object classification module mayinclude the systems and use the methods described in commonly owned U.S.patent application Ser. No. 12/397,287 entitled “Dynamic ObjectClassification,” the entire contents of which are incorporated herein byreference. The temporal object classification module 208 may aggregatethe classifications made by the object classification module 210.

The video analytics module 200 also includes an object indexing module212 connected to the storage system 110. The object indexing module 212is operable to generate signatures for objects. The signatures may bestored in the metadata database 112 and may act as index elements forvideo images of the objects. The object indexing module 212 is describedin more detail below.

The video analytics module 200 also includes an object search module 214connected to the storage system 110 and the user interface 104. Theobject search module 214 is operable to search through signatures storedin the storage system 110 to identify whether an object was present inpreviously captured images. The object search module 214 is described inmore detail below.

The video analytics module 200 also includes an object categorizationmodule 216 connected to the object classification module 210. The objectcategorization module 216 is operable to prioritize tracking whenmultiple objects are detected in an image. For example, the objectcategorization module 216 is operable to use classification informationgenerated by the object classification module 210 to establish an orderfor attempting to associate objects of a current frame with objectsdetected in past frames. The object categorization module 216 isdescribed in more detail below.

Data generated by the video analytics module 200 may be used by a rulesengine 220 to determine whether one or more user-specified rules havebeen violated. For example, the rules engine 220 may trigger an alarmthat is presented on the display 114 of the user interface if a human isdetected in the field of view of one of the image capturing devices 102.

As used herein, the term “module” is a component that may comprise oneor more hardware circuits or devices and/or one or more softwareroutines, functions, object or the like. A module may also be entirelyhardware, entirely software, comprise firmware, or comprise somecombination of the foregoing. As used herein, the term “system” refersto a tangible thing.

Match Classifier

The video analytics module 200 also includes a match classifier 218connected to the object tracking module 206, the object indexing module212, and the object search module 214. The match classifier 218 isoperable to receive an input pattern z representing signatures of twoobjects and determine whether the signatures match (e.g., whether thesignatures are sufficiently similar). The match classifier 218 may beused by the object tracking module 206, the object indexing module 212,and the object search module 214 to assist the modules with theirvarious operations. Thus, a unified framework for tracking, indexing,and searching may be achieved via the match classifier 218.

The match classifier 218 is operable to match objects based upon thesignatures (e.g., features or appearance characteristics) of theobjects. For example, the match classifier 218 receives data (i.e., theinput pattern z) related to the signatures of two objects captured indifferent frames and determines whether the objects correspond to thesame physical object.

A signature of an object may be made of a subset of features {circumflexover (F)}={f_(k1), f_(k2), . . . , f_(km)} selected from a set offeatures F={f₁, f₂, f₃, . . . f_(n)}. The elements of {circumflex over(F)} may be viewed as some transformation of an image region R of anobject. Thus, a signature x of an object may take on the following form:

$\begin{matrix}{x = \begin{pmatrix}{{\hat{f}}_{1} = {f_{k\; 1}(R)}} \\{{\hat{f}}_{2} = {f_{k\; 2}(R)}} \\\vdots \\{{\hat{f}}_{m} = {f_{km}(R)}}\end{pmatrix}} & (1)\end{matrix}$

The features {circumflex over (f)}₁, {circumflex over (f)}₂, . . . ,{circumflex over (f)}_(m) of an object may correspond to a number ofappearance characteristics such as, but not limited to, aspect ratio,hue, intensity, edge orientations, texture, corner features, raw imagepixel data, normalized saturation, and localized descriptors such asScale-invariant feature transform (SIFT) features. The features includeboth color features (e.g., hue and saturation) and gray-scale features(e.g., intensity and edge orientation). This allows the match classifier218 to work with both color and black and white images. An image of anobject may be divided into regions or subwindows, and the features{circumflex over (f)}₁, {circumflex over (f)}₂, . . . , {circumflex over(f)}_(m) may correspond to the appearance characteristics of the objectin the regions or subwindows. Moreover, the features {circumflex over(f)}₁, {circumflex over (f)}₂, . . . , {circumflex over (f)}_(m) mayrepresent feature vectors (e.g., histograms in which the histogram binscorrespond to vector components) of the appearance characteristics andmay be used by the match classifier 218 to determine whether objectsmatch. For example, histograms of the edge orientations of an object maybe constructed for different regions (e.g., subwindows) of the object'simage. In other words, an image of an object may be divided intosubwindows, and edge orientations may be calculated for each pixel ofthe subwindows. The edge orientation of a pixel may be derived using asteerable filter (e.g., using a Gaussian derivative filter in multipledirections). For example, an image may be convolved with a firstderivative of a Gaussian distribution oriented at zero and 90 degrees,which may be used as basis filters to synthesize numerous orientations.Gaussian derivative responses may be computed at eight orientations foreach pixel, and for each pixel, the orientation with the maximumresponse may be chosen as the direction for the pixel. Using a steerablefilter allows dominant directions to be assigned to the pixels of asubwindow, and allows a histogram of the directions to be constructedfor the subwindow. For example, for a given pixel, a steerable filtermay be used in multiple directions to generate multiple responses, andthe direction corresponding to the maximum directional derivativeresponse is assigned as the direction of the pixel. A signature x of anobject may capture the uniqueness of the object while allowing forappearance variations among different instances of the object. Althoughnot required, a signature of an object may be computed with maximumresolution available for the camera system 100 and may not be affectedby post-facto down-sampling or compression.

FIG. 3 is a block diagram showing a method 300 that may be used togenerate an input z for the match classifier 218 based on a signature xfrom a first object and a signature y from a second object. Generally,the signatures x and y from the first and second object are combined toform the input z (step 302). For example, feature {circumflex over (f)}₁of signature x may be combined with feature {circumflex over (f)}₁ ofsignature y. The signatures x and y may be combined mathematically usingcombination functions such as a histogram intersection or aconcatenation of features. A combination function may be thought of as adistance measure d between the features. Many different distancemeasures may be used to come up with the input z such as an L₁ distance,a Manhattan distance, an L₂ distance, and a Bhattacharyya distance.Thus, many combinations or distance measures are possible for feature{circumflex over (f)}₁ of the signature x and feature {circumflex over(f)}₁ of signature y, for example. The distance measures may correspondto a measure of similarity between the features; instances of the sameobject may be “close” in feature space (e.g., small distance measure),while instances of different objects may be “far away” in feature space(e.g., large distance measure). When combining edge orientationhistograms from the signatures x and y, for example, concatenation ofthe edge orientation histograms may serve as a distance measure d. Theinput z is supplied to the match classifier 218, and the matchclassifier 218 makes a decision as to whether the first and secondobject match. As described below, a training process may automaticallychoose the best features of the signatures x and y, together with thebest combination or distance measure, to achieve a high level ofaccuracy in matching objects. The output of the match classifier 218 maycorrespond to a decision step value s(z) as described below. Thedecision step value s(z) may indicate whether the first and secondobject match, and may include a value corresponding to a confidencelevel in its decision.

The match classifier 218 may be represented by a classification problemdefined generally by a classifier function Γ(z), in which two objectsrepresented by the input pattern z are declared a match when Γ(z)>0 oras a non-match when Γ(z)<0. Generally the classifier function Γ(z) isparameterized with a set of parameters, and the input z is composed of acombination of the features described above. The magnitude of the outputof the classifier function may reflect the level of confidence of thematch classifier 218 in its decision. This level of confidence mayconstitute a measure of similarity (or dissimilarity).

An embodiment of the structure of the match classifier 218 will now bedescribed in more detail with reference to FIGS. 4-6. The matchclassifier 218 includes multiple steps 400 (N decision steps) configuredin a cascade configuration (i.e., step 1, followed by step 2, followedby step 3, . . . , followed by step N) as shown in FIG. 4. The N stepsoperate to determine whether two signatures match. Unlike a knowncascade classification system proposed in Paul Viola & Michael Jones,“Robust Real-Time Face Detection,” International Journal of ComputerVision, pp. 137-154 (2004), each of the initial N−1 steps of the presentembodiment is operable to make one of three decisions: (1) accept twoobjects as a match, (2) reject the objects as a match (i.e., declare anon-match), and (3) forward the decision onto the next step. Forexample, the input z is supplied to step 1, and step 1 decides whetherto (1) accept the objects as a match, (2) reject the objects as a match,or (3) forward the decision to step 2. The decision to accept, reject orforward is based on a value produced by the step (i.e., the decisionstep value s(z)). The final or Nth step is operable to either (1) acceptthe objects as a match or (2) reject the objects as match.

FIG. 5A is a block diagram showing one of the initial N−1 steps 400 inmore detail. Each step 400 includes one or more stages 500. For each ofthe stages 500, a combination (e.g., a distance measure orconcatenation) of one of the features {circumflex over (f)}₁,{circumflex over (f)}₂, . . . , {circumflex over (f)}_(m) from a firstobject and a corresponding one of the features {circumflex over (f)}₁,{circumflex over (f)}₂, . . . , {circumflex over (f)}_(m) from a secondobject is extracted (represented by blocks 502) from the input z andsupplied to its corresponding stage 500. For example, a combination ofthe feature {circumflex over (f)}₁ from the first object and the feature{circumflex over (f)}₁ from the second object may be supplied to thefirst stage 500. Each stage 500, therefore, has a corresponding featurecombination associated with it. The stage/feature combination may bepredetermined by a learning algorithm during a training process asdescribed below. Moreover, a feature combination supplied to the firststage (stage 1) of FIG. 5A, for example, may be different from or thesame as the feature combination supplied to the second stage (stage 2).

In general, a stage 500 may be represented by a stage function g whichis defined as follows:g(x,y)→ygεG(x,y)εz andγε[−1,1]  (2)in which G represents a general set from which the specific stagefunction g is chosen and (x,y) represents a combination of a featureextracted from signature x and a feature extracted from signature y. Asdescribed above, a combination of features supplied to a stage 500 maybe considered a distance measure d. The set G may, therefore, be writtenas G=D×T where D is a set of all possible distance measures calculatedat step 302 of FIG. 3 (e.g., dεD, d: (x,y)→z) and T represents a set ofpossible transformations (i.e., mappings) such that for tεT, t: z→y.Thus, the stage function g may take on the following expanded form:g _(i)({circumflex over (f)} _(k) ^(x) ,{circumflex over (f)} _(k)^(y))=t(d({circumflex over (f)} _(k) ^(x) ,{circumflex over (f)} _(k)^(y)))  (3)

A stage 500 represents a discriminant function that includes a weightvector {right arrow over (w)} and an activation function β. As notedabove, the features {circumflex over (f)}₁, {circumflex over (f)}₂, . .. , {circumflex over (f)}_(m) may represent feature vectors.Additionally, a distance measure d between feature vectors may representa distance vector {right arrow over (d)}. The stage function may berewritten as follows:g _(i)({right arrow over (d)} _(k))=β_(i)({right arrow over (w)} _(i)·{right arrow over (d)} _(k))  (4)where {right arrow over (w)}_(i)·{right arrow over (d)}_(k) representsthe inner product of the weight vector {right arrow over (w)} and thedistance vector {right arrow over (d)}_(k). The activation functionβ_(i) may be any function such as, but not limited to, a sigmoidfunction or a radial basis function. The activation function β_(i) isused to map the inner product of the weight vector {right arrow over(w)} and the distance vector {right arrow over (d)}_(k) to a valuebetween zero and one. The scalar value γ may be determined bycalculating a differential of the discriminant function. Unlike knowncascade classifiers that include the same discriminant function for allstages, the discriminant functions for the stages of the presentembodiment may be different from each other. Moreover, the weight vector{right arrow over (w)}_(i) and activation function β_(i) for each stage500 are automatically determined during training as described below.

Each step 400 is a linear combination (represented by scale and summingblock 504) of one or more of the stages 500. In other words, the scalarvalues y of the stages 500 are scaled and summed to produce the decisionstep value s(z). In general, a step function may be representedmathematically as follows:s(z)=Σ_(∀) _(i) α_(i) g _(i)(x _(k) ,y _(k))  (5)

-   -   where Σ_(∀) _(i) α_(i)=1        Because s(z) (i.e., the decision step value) is a convex        combination of stages, s(z) and g(x,y) have the same range.        Weighting coefficients α are chosen by a learning algorithm        during training. The decision step value is compared        (represented by block 506) to one or both of an acceptance        threshold τ_(α) and a rejection threshold τ_(r) to determine        whether two objects match, to reject the objects as a match, or        to forward the decision to the next step 400. The comparison may        be represented as follows:        Accept if τ_(α) <s(z)≦1        Forward if τ_(r) <s(z)≦τ_(α)        Reject if −1<s(z)≦τ_(r)  (6)

FIG. 5B illustrates one example of the acceptance and rejectionthresholds in the interval [−1,+1]. The acceptance threshold τ_(α) andthe rejection threshold τ_(r) are chosen during training by a learningalgorithm based on user-specific false positive and false negativerates. Each step 400 may have values for τ_(α) and τ_(r) that aredifferent from or the same as the values the other steps 400 have. Adecision to “Accept” implies that the classifier is confident that twoobjects match. A decision to “Forward” implies that the classifier isunsure and defers the decision to the next step. A decision to “Reject”implies that the classifier is confident that two objects do not match.At any step, if the decision is not to “Forward” to the next step 400,the accept/reject decision is made at that point and the evaluation iscomplete. The decision step value s(z) at that step is the output valuefor the match classifier 218. The decision step value may correspond toa decision confidence level for the corresponding step 400. For example,a decision step value close to one may represent that the correspondingstep 400 is more confident that two objects match compared to a decisionstep value slightly above the acceptance threshold τ_(α). Alternatively,an increase in the decision step value may not necessarily correspond toa higher decision confidence level (i.e., a probability that thedecision was correct). The confidence level associated with eachdecision step value may be empirically estimated during training asobjects are correctly and incorrectly matched with different decisionstep values. The confidence levels of the decision step values aredescribed in more detail below. As noted above, the final stage (stageN) in the match classifier 218 is forced to always accept or reject:Accept if 0<s(z)≦1Reject if −1<s(z)≦0  (7)

FIG. 6 is flow chart depicting an operational flow 600 of the matchclassifier 218 by way of example. First, the signatures of a first andsecond object are combined (step 602). The first and second objects maycorrespond to the same physical object. Step 602 correlates with step302 of FIG. 3. The input z is transmitted to the first step 400 of thematch classifier 218 (step 604). The combined features (e.g., thedistance measures) that correspond to the first through S^(th) stages500 are identified in and chosen from the input z (step 606).Alternatively, rather than transmitting the input z to the first step400, the combined features used by the stages 500 of the first step 400may be selected and only those combined features may be transmitted tothe first step 400. The combined features are supplied to theirrespective stages 500, and the stages 500 map the combined features toscalar values y (step 608). The scalar values are scaled (i.e.,weighted) and summed to produce a decision step value s(z) (step 610).The decision step value is compared to one or more of the acceptancethreshold τ_(α) and the rejection threshold τ_(r) (step 612). If thedecision step value is greater than the acceptance threshold τ_(α), thefirst and second objects are accepted as a match (step 614). If thedecision step value is less than or equal to the rejection thresholdτ_(r), the first and second objects are rejected as a match (step 616).If the decision step value is greater than the rejection threshold τ_(r)but less than or equal to the acceptance threshold τ_(α), the input z isforwarded to the second step 400 (or, in the alternative, only thosefeature combinations used by the second step 400 are transmitted to thesecond step 400) (step 618). The first and second objects may beaccepted or rejected as a match at any step 400 within the cascade.

Training the Match Classifier

A method of training the match classifier 218 will now be described.Conventional classifiers may be trained with the learning algorithmAdaBoost or some variant of AdaBoost. While AdaBoost has proved itsworth in some applications, the algorithm and objective function used inthat learning process have some limitations. For example, for AdaBoostto be effective, distance measures of matching and non-matching objectscannot overlap each other significantly in feature space. In otherwords, the features of the objects should separate a classificationspace well. Moreover, because AdaBoost uses weak learners, a largecollection of weak learners may be necessary to form a full classifiercapable of achieving a desired accuracy.

According to one embodiment, an alternative objective function andlearning algorithm called Sequential Discriminant Error Minimization(SDEM) is used to train the match classifier 218. SDEM was proposed inSaptharishi, “Sequential Discriminant Error Minimization: The Theory andits Application to Real-Time Video Object Recognition,” (Carnegie MellonUniversity, 2005), the entire contents of which are incorporated hereinby reference. SDEM can deal with features or distance measures that donot necessarily separate a classification space well. Unlike AdaBoostand other similar boosting techniques, SDEM may use weak learners orother discriminant functions that are not necessarily considered weak.Thus, the number of steps 400 and stages 500 of an object classifier maybe significantly less using SDEM for training compared to AdaBoost. Fora given feature, SDEM is often capable of learning the best classifierfor the corresponding feature space. The best features of objects,distance measures, and transformations may be automatically selected fora given classification problem.

Generally, the SDEM algorithm is used to train the combination of stages500 for each step 400. As defined in equation (3), a stage 500 includesa stage function g_(i)(x,y) that is equal to a transformation t of adistance measure d of features {circumflex over (f)}_(k) ^(x) and{circumflex over (f)}_(k) ^(y) (i.e., t(d({circumflex over (f)}_(k)^(x), {circumflex over (f)}_(k) ^(y)))). The training task chooses thebest transformation t, the best distance measure d, and the best feature{circumflex over (f)}_(k), such that when a particular stage 500 isadded to a step 400, the performance of the object classifier ismaximized. In other words, the SDEM algorithm selects the transformationt, distance measure d, and the feature {circumflex over (f)}_(k) for aparticular stage so as to maximize an objective function. The set F offeatures and the set of distance measures D may be finite, and the set Tof transformations may be continuous and differentiable. Following theSDEM algorithm, for each choice of a discrete pair ({circumflex over(f)}_(k),d_(j)), a search is performed in the set T to identify thetransformation t that performs best on a training data set. The searchin the set T may be performed using standard unconstrained optimizationtechniques such as, but not limited to, a Quasi-Newton optimizationmethod. Once the best transformation t is identified for each choice of({circumflex over (f)}_(k),d_(j)), the best feature/distance measurepair may be selected according to an estimated generalization error{circumflex over (ε)}(g). Selection of the best feature/distance measurepair may be written as follows:g({circumflex over (f)} _(k) ^(x) ,{circumflex over (f)} _(k)^(y))=min_(∀(k,j),∀tεT){{circumflex over (ε)}(t(d _(j)({circumflex over(f)} _(k) ^(x) ,{circumflex over (f)} _(k) ^(y))))}  (8)

When each stage 500 is added to a step 400, there may be n uniquefeature types and b unique distance measures. Thus, a total of n×b pairs({circumflex over (f)}_(k),d_(j)) may be explored when adding a newstage 500. One of the properties of the SDEM algorithm is that when astage 500 is added to a step 400, the addition of the stage 500 improvesthe performance of the object classifier on the training data set. If anew stage cannot be identified that improves the performance of theobject classifier, the SDEM algorithm automatically terminates.Alternatively, rather than waiting for SDEM algorithm to automaticallyterminate, a number of stages 500 of a step 400 may be determined by auser. In other words, the SDEM algorithm terminates training when amaximum number of stages 500 set by the user is reached or when no stage500 can be added that will improve performance.

The SDEM algorithm selects a series of features, distance measures, andtransformations such that when combined, the series outperforms a singleone of the features/distance measures/transformations. For example,while the aspect ratio of an object might be a poor feature to use formatching, when combined with local gradient information, the aspectratio may improve the matching accuracy corresponding to the localgradient information alone. A number of simple features, distancemeasures, and transformations may be combined to create a highlyaccurate match classifier 218. The training task creates super-featuresby combining a set of appearance features and distance measures for twoobjects.

The training task for building the stages 500 of first step 400 will nowbe described in more detail with reference to FIGS. 7 and 8. Thefollowing description is also applicable to the stages 500 of the othersteps 400. FIG. 7 is a block diagram of a method 700 of generatingtraining data. A sequence of images of objects may be presented to auser for verification as to whether the images correspond to the sameobject or different objects. For example, two images may besimultaneously presented, or sequentially presented, to the user forverification. When the images correspond to the same object, a matchexample is created (step 702). When the images correspond to differentobjects, a non-match example is created (step 704). Distance measuresd₁, d₂, . . . , d_(m) are used for the features of the match andnon-match examples. The features {circumflex over (f)}₁, {circumflexover (f)}₂, . . . , {circumflex over (f)}_(k) of the objects of thematch and non-match examples may have more than one distance measureassociated with it. For example, b different distance measures may beavailable for one or more of the features {circumflex over (f)}₁,{circumflex over (f)}₂, . . . , {circumflex over (f)}_(k). Thus, d₁ andd₂ may both correspond to feature {circumflex over (f)}₁ of the objects.The features {circumflex over (f)}₁, {circumflex over (f)}₂, . . . ,{circumflex over (f)}_(k) of the match and non-match examples arecombined by using one or more of the different distance measuresassociated with the particular features. For example, if d₁ correspondsto {circumflex over (f)}₁, d₁ is used to combine the feature {circumflexover (f)}₁ of one object of a match example with the feature {circumflexover (f)}₁ of the other object of the match example. The features{circumflex over (f)}₁ of the objects of the other match and non-matchexamples may also be combined. After a number of match and non-matchexamples are created and the features {circumflex over (f)}₁,{circumflex over (f)}₂, . . . , {circumflex over (f)}_(k) of the objectsof the examples are combined using the appropriate distance measures d₁,d₂, . . . , d_(m), the examples and combinations may be used in atraining algorithm to train the match classifier 218 (step 706).

FIG. 8 is a flow chart of a training method 800 that may be used totrain the match classifier 218. The combinations of features {circumflexover (f)}₁, {circumflex over (f)}₂, . . . , {circumflex over (f)}_(m) ofthe objects of the match and non-match examples (i.e., featurecombinations 802) are identified and associated with the distancemeasures (e.g., d₁, d₂) that were used to combine the features (steps804 a, 804 b, 804 c). For example, for the distance measure d₁, thecombinations of the appropriate feature {circumflex over (f)} formed byusing d₁ are identified (step 804 a).

After the feature combinations 802 corresponding to the distancemeasures d₁, d₂, . . . , d_(m) are identified, the best transformation tis selected for each of the distance measures d₁, d₂, . . . , d_(m)(steps 806 a, 806 b, and 806 c). Because a particular feature{circumflex over (f)} is associated with each distance measure, the besttransformation is selected not only for the distance measure, but alsofor the feature associated with the distance measure. Thetransformations may be selected based on standard optimizationtechniques. A transformation t may be viewed as a decision boundary thatseparates the feature combinations 802 of the match and non-matchexamples. Thus, the best transformation t corresponds to a decisionboundary that best separates the match and non-match examples for thecorresponding feature {circumflex over (f)} and distance measure. In thecontext of the discriminant function made up of the weight vector ŵ_(i)and the activation function β_(i), selection of the best transformationt corresponds to selection of the activation function β_(i) and thecomponents of the weight vector {right arrow over (w)}_(i) that bestseparate the match and non-match examples. The activation function β_(i)may be selected from a set of multiple function types such as, but notlimited to, a sigmoid function and a radial basis function (e.g., aGaussian function). Thus, unlike known cascade classifiers, a firststage may include a different type of discriminant function compared toa second stage.

After the transformations for the distance measures d₁, d₂, . . . ,d_(m) are selected, a value of an objective function corresponding toeach distance measure/transformation combination is calculated (steps808 a, 808 b, and 808 c). The objective function may be proportional toa measure of errors (e.g., declaring a match a non-match or vice versa)or it may be a nonlinear but monotonically varying function of theclassification errors. The calculated values of the objective functionmay be related to the number and/or severity of classification errorsmade by the different distance measure/transformation combinations. Forexample, a first calculated value may be related to the number ofclassification errors made by the distance measure d₁ and itscorresponding transformation. The calculated values of the objectivefunction are compared, and the distance measure/transformationcombination that has the maximum calculated value is selected for thefirst stage 500 of the first step 400 (step 810).

After the distance measure (and its associated feature) andtransformation are selected for the first stage 500, the featurecombinations 802 of the different match and non-match examples areweighted with different weights taking into account the decisions madeby the first stage 500 (step 812). Each of the feature combinations 802of the match and non-match examples may be thought of as correspondingto data points in function space. The feature combinations 802 of thematch and non-match examples may be weighed as a function of how closetheir corresponding data points are to the decision boundary representedby the first stage 500. For example, the feature combinations 802corresponding to data points close the decision boundary in featurespace may be weighed with higher weights compared to featurecombinations 802 with data points far away from the decision boundary sothat a second stage 500 may be trained by focusing more on those featurecombinations 802 that somewhat confused the first stage 500. Thedistance between a data point and the decision boundary of the firststage 500 may be related to the scalar value γ calculated for theexample corresponding to the data point.

After the feature combinations 802 of the examples are weighted, thebest transformations are again selected for the distance measures d₁,d₂, . . . , d_(m) (steps 806 a, 806 b, and 806 c are repeated). Thefeature combinations 802 corresponding to the distance measures d₁, d₂,. . . , d_(m) however, are now weighted, and the best transformation tfor each of the distance measures d₁, d₂, . . . , d_(m) is selectedtaking into account the first stage 500. The best transformation t maycorrespond to the transformation that results in the greatest increaseof the objective function value. The values of the objective functionare again calculated and compared to determine the distancemeasure/transformation for the second stage 500 (steps 808 a, 808 b, 808c, and 810 are repeated). To build a third stage 500, the featurecombinations 802 of the match and non-match examples are re-weighted, inwhich the examples that were somewhat confusing to the first and secondstages 500 are given higher weights. Again, steps 806 a, 806 b, and 806c are repeated, but feature combinations 802 corresponding to thedistance measures d₁, d₂, . . . , d_(m) are now re-weighted, and thebest transformation t for each of the distance measures d₁, d₂, . . . ,d_(m) is selected taking into account the first and second stages 500.The values of the objective function are again calculated and comparedto determine the distance measure/transformation for the third stage 500(steps 808 a, 808 b, 808 c, and 810 are repeated). The process ofchoosing the best distance measure, transformation, and weighting thefeature combinations 802 of the examples through each iteration oftraining a new stage may be thought of as a gradient ascent in functionspace, or as a process to increase the total value of the objectivefunction.

Once the first step 400 is trained, the thresholds τ_(α) and τ_(r) areselected such that desired false positive and false negative rates maybe attained. Moreover, as the stages 500 are constructed for the firststep 400, the weighting coefficients α are also selected. For example,as each stage 500 is added to the first step 400, the weightingcoefficients α for the stages are adjusted to find the values for theweighting coefficients α that correspond to the lowest overall errorrate for the first step 400. For example, the weighting coefficients αmay be selected by using a line search optimization strategy.

After the first step 400 is trained, the stages 500 of a second step 400may be trained. However, fewer than all of the match and non-matchexamples used to train the first step 400 may be used to train thesecond step 400. For example, only those match and non-match examplesthat the first step 400 could neither accept nor reject may be used totrain the second step 400. In other words, the second step 400 istrained with the examples that had a corresponding decision step valuegreater than the rejection threshold τ_(r), but less than or equal tothe acceptance threshold τ_(α). This allows the second step 400 to focusonly on those examples that the first step 400 found confusing.

Once the match classifier 218 is trained with the examples, the matchclassifier 218 may continue through other training steps to refine thedistance measures/transformations selected for the different stages 500.A high-level approach to training the match classifier 218 is shown inthe flow chart of FIG. 9, which illustrates a training method 900. Imagedata 901 (e.g., raw video data) is supplied to a simple base or seedsystem that is capable of basic detection, tracking, and classificationof objects. The base system detects, tracks, and classifies objectsrepresented in the image data 901 and generates metadata correspondingto the objects (step 902). The base system selects a set of objects thatare detected and tracked (step 904). The selection of the objects maydepend on the amount of time an object was in the field of view of animage capturing device 102, or may depend on how confident the basesystem was in its classification of an object. Other rules may bespecified for dictating whether an object is selected by the basesystem.

Images of the objects selected by the base system are presented on adisplay to a user so the user can manually label the objects as match ornon-match examples. The user manually labels the objects, and thelabeled examples are supplied to the match classifier 218 being trained(step 906). Feature combinations may be calculated for the labeledexamples using the distance measures d₁, d₂, . . . , d_(m). The featurecombinations of the manually labeled examples may correspond to thefeature combinations 802 described above. The match classifier 218 istrained with the feature combinations 802 such as according to thetraining method 800 as described above with reference to FIG. 8 (step800). Image data 901 are supplied to the match classifier 218. The matchclassifier 218 determines whether objects represented in the image data901 match and generates metadata representing matched and non-matchedobjects (step 910). Each match and non-match generated by the matchclassifier 218 has a match confidence level associated with it. Thematch confidence level corresponds to the decision step value of thestep 400 that declared the match or non-match. The match confidencelevels generated by the match classifier 218 are analyzed to identifyinstances of objects that were confusing to the match classifier 218(e.g., matches and non-matches with a low match confidence levels). Theperformance of the match classifier 218 is evaluated to determinewhether the match classifier's performance is acceptable (step 912).

To determine whether the match classifier's performance is acceptable, adisjoint test set may be used in which matches and non-matches of thedisjoint test set are known prior to classification by the matchclassifier 218. The image data 901 supplied to the match classifier 218may correspond to the disjoint test set, and the classifications made bythe trained object classifier may be compared to the actual matches andnon-matches. From this comparison, the performance of the matchclassifier 218 may be determined. If the performance is not equal to orabove some predefined performance level, the confusing objects arepresented to the user for manual labels (step 904). The user labels theconfusing objects and the new labeled matches and non-matches are usedto retrain the match classifier 218 (steps 906 and 908). When the matchclassifier 218 is retrained, the distance measure/transformationcombinations for the different stages 500 may be updated based on thenew labeled examples. The retrained match classifier is used to classifyobjects represented in the image data 901 and the performance of theretrained match classifier is evaluated (steps 910 and 912). Theretraining process may continue until the performance of the matchclassifier 218 is acceptable. When the performance of the matchclassifier 218 is acceptable, it may be deployed (step 914). Thetraining process may be recast into the following steps:

-   -   1. Manually label some small fraction of a dataset.    -   2. Train a match classifier 218 using the fraction of the        dataset.    -   3. Use a newly trained classifier to automatically label the        complete dataset.    -   4. Select a set of automatically labeled data points that were        confusing to the match classifier 218.    -   5. Manually label the confusing data points.    -   6. Repeat the training with all the new labeled data points.    -   7. Go to step 3.

Given the steps-of stages architecture of the match classifier 218, anew step may be added to the match classifier 218. This new step may betrained to correct mistakes made by the match classifier 218.Alternatively, a new step may be trained to replace the last or N^(th)step of the match classifier 218. After the match classifier 218 istrained by the process described above, the match classifier 218 mayperform various operations such as tracking, generating indexingelements, and performing object-based (e.g., appearance-based) searches.In other words, the match classifier 218 need not be trained separatelyfor each of its various operations.

The manner in which the match classifier 218 operates to match objectinstances has been determined at least in part by a training orself-learning process that automatically configures the match classifier218 using a set of possible object features. That configuration mayinclude one or more of the following determinations: (1) selecting whichfeatures from the set of possible features to use; (2) determining theorder in which the selected features are evaluated; (3) determining howmultiple features are combined (i.e., the weighting coefficients used tocombine multiple variables); (4) selecting distance measures from a setof available distance measures (e.g., L₂, Manhattan, Bhattacharyya).Other aspects of the match classifier 218 may be configured during thetraining process. The training process may take place off-line prior todeployment of the match classifier 218 and/or on-line during on-lineoperation of the match classifier 218.

Match Confidence

The decision step value s(z) is correlated with the match classifier'sestimate as to how similar it thinks two objects are (e.g., matchconfidence). The correlation may not be linear, i.e., the step 400 maygenerate a high positive value, but the objects may not match.Typically, by virtue of the training process, the higher the value ofs(z), the less likely that the step 400 made a mistake. The confidencelevel associated with a certain value of s(z) may be calculated by firstdefining an indicator function ε(Γ(z)) in which:

$\begin{matrix}{{ɛ( {{\Gamma(z)} = {s(z)}} )} = \{ \begin{matrix}{0,{{{Instances}\mspace{14mu}{match}\mspace{14mu}{and}\mspace{14mu}{s(z)}} > 0}} \\{0,{{{Instances}\mspace{14mu}{do}\mspace{14mu}{not}\mspace{14mu}{match}\mspace{14mu}{and}\mspace{14mu}{s(z)}} \leq 0}} \\{1,{{{Instances}\mspace{14mu}{match}{\mspace{11mu}\;}{and}\mspace{14mu}{s(z)}} \leq 0}} \\{1,{{{Instances}{\mspace{11mu}\;}{do}\mspace{14mu}{not}\mspace{14mu}{match}\mspace{14mu}{and}\mspace{14mu}{s(z)}} > 0}}\end{matrix} } & (9)\end{matrix}$A confidence function, Ψ(Γ(z)), may be defined as the probability thatthe step 400 declared two instances to be a match, and that it wascorrect for an output of s(z)=v. Thus, for a small quantization interval[v−Δ, v+Δ], the confidence function may be expressed as follows:Ψ(Γ(z))=P _(ε,Ω|Γ)(ε(Γ(z))=0,ω=match|s(z)<v+Δ)−P_(ε,Ω|Γ)(ε(Γ(z))=0,ω=match|s(z)<v−Δ)  (10)

Note, it may be considered that the step 400 declares two instances tobe a match when s(z)>0, i.e., P_(,Ω|Γ)(ω=match|s(z)>0)=1. Thus, for v>0,equation (10) can be expressed as:Ψ(Γ(z))=P _(ε|Ω,Γ)(ε(Γ(z))=0|ω=match,0<s(z)≦v+Δ)−P_(ε|Ω,Γ)(ε(Γ(z))=0|ω=match,0<s(z)<v−Δ)  (11)Equation (11) represents the true positive rate when vε[Δ,1−Δ] ands(z)ε[v−Δ,v+Δ].

Similarly, the confidence of a step 400 in declaring that the objectbelongs to the negative class for v≦−Δ may be expressed as:Ψ(Γ(z))=P _(ε|Ω,Γ)(ε(Γ(z))=0|ω=no match,v+Δ≦s(z)≦0)−P_(ε|Ω,Γ)(ε(Γ(z))=0|ω=no match,v−Δ<s(z)≦0)  (12)Equation (12) represents the true negative rate when vε[−1+Δ, −Δ] ands(z)ε[v−Δ,v+Δ]. Thus, if the probability (as defined in equations (11)and (12)) that the step 400 is correct for any observed output values(z)=v is high, then the step 400 is considered to be confident in itsanswer. For this self assessment of confidence, a probability measure{circumflex over (p)}_(ε|Ω,Γ) is estimated from the training examplesand the confidence function Ψ(Γ(z)) is inferred. If the confidencefunction Ψ(Γ(z)) is less than a critical confidence threshold ψ_(c) fora given output value for s(z), then the step 400 is considered to bediffident or confused for that output value. Diffident classificationsare forwarded to the next step 400. Thus, the confidence functionΨ(Γ(z)) may be used during training to identify the objects that areconfusing to an match classifier 218. If discriminant functions thatconstitute the stages 500 are a good approximation to the Bayes optimaldecision boundary, then the decision step value s(z) will bemonotonically related to the confidence function Ψ(Γ(z)). For theinitial steps 400, the object classifier may not approximate the Bayesdecision boundary well. Thus, the decision step value s(z) for a givenstep 400 and the confidence function Ψ(Γ(z)) may not always bemonotonically related.

The confidence function Ψ(Γ(z)) may be used to determine the acceptancethreshold τ_(α) and the rejection threshold τ_(r) for the differentsteps 400. As opposed to other cascade classifier architectures, if theconfidence function Ψ(Γ(z)) for matches is monotonically increasing withthe decision step value s(z), the acceptance threshold τ_(α) may bechosen such that true positive constraints are met. If the confidencefunction Ψ(Γ(z)) does not increase monotonically with the decision stepvalue s(z), then the acceptance threshold τ_(α) may remain saturated at1, i.e., no instances are accepted as matches in the corresponding step400. The lack of monotonicity indicates that in the match region, thedecision boundary does not reflect the Bayes optimal classifiersufficiently well. Similarly, the rejection threshold τ_(r) is chosen ifthe non-match confidence is monotonically related to the decision stepvalue s(z). In practice, the non-match class may be far more denselypopulated in feature space than the match class. Thus, while themonotonic relationship may not exist for the match class in early steps,it most likely exists for the non-match class in the early steps. Thematch confidence level corresponding to the match classifier 218 is theclassification confidence associated with the decision step value s(z)generated by the step 400 that either accepted or rejected a match.

Tracking

Tracking may be thought of as locating an object in each video frame orimage, and establishing correspondences between moving objects acrossframes. Tracking may be performed within a single image capturing device102 or across multiple image capturing devices 102. In general, theobject tracking module 206 may use object motion between frames as a cueto tracking, while also relying on the match classifier 218 fortracking. Using the match classifier 218 for tracking may enableaccurate tracking even when some occlusion is present, motion issomewhat erratic, and a fair number of crisscrossing objects are presentin the field of view of an image capturing device 102.

FIG. 10 is a block diagram of a system 1000 used by the object trackingmodule 206 for tracking objects. The object tracking module 206 includesa motion modeling module 1002 that is operable to estimate changes oflocation of an object between sequential images or frames of a field ofview of an image capturing device 102. The motion modeling module 1002may predict the location and size of an object in a new frame based uponits previously estimated trajectory and velocity. The motion modelingmodule 1002 may be used to rank observations of objects in a currentframe by their likelihood or probability of corresponding to aparticular object that is being tracked. The object tracking module 206may generate a list of tracked objects or targets that have beendetected in previous frames. For example, the list of tracked objectsmay include ten objects, the current frame may include five detectedobjects, and a ranking (e.g., one through five with one being thehighest ranking) of the five detected objects may be generated for eachof the ten tracked objects based upon match probability information. Theobject tracking module 206 may generate an identification number oranother type of identification symbol for each object that it tracks.The motion modeling module 1002 may use a Kalman filter to predict theposition and velocity of an object.

After the motion modeling module 1002 ranks objects of the current frameby their likelihood of corresponding to tracked objects, the objecttracking module 206 may call on the match classifier 218 to determinewhether an object of the current frame corresponds to a tracked object.For a given tracked object, the object tracking module 206 may call onthe match classifier 218 to first determine whether the highest rankedobject of the current frame matches the tracked object. Moreover, theobject tracking module 206 may use match probability information todetermine an order for the tracked objects. For example, if the motionmodeling module 1002 determines that the probability of a match betweena first tracked object and its highest ranked object is greater than theprobability of a match between a second tracked object and its highestranked object, then the object tracking module 206 may call on the matchclassifier 218 to attempt to match the first tracked object with itshighest ranked object before attempting to match the second trackedobject with its highest ranked object. Although the motion modelingmodule 1002 has been described as being used prior to the matchclassifier 218, the motion modeling module 100 may be used before orafter the match classifier 218. In general the motion modeling module1002 may be used in conjunction with the match classifier 218. Themotion modeling module 1002 may be used to help determine the confidenceof a match or non-match declared by the match classifier 218. Forexample, the match classifier 218 may declare that two objects are amatch, but the motion modeling module 1002 may determine that the twoobjects are too far apart in the field of view to correspond to the sameobject. In that case, the motion modeling module 1002 may be used toincrease the acceptance threshold τ_(α). The match confidence levelassociated with the match classifier 218 may then be analyzed inconjunction with determinations made by the motion modeling module 1002to declare a match or non-match.

FIG. 11 is a flow chart of a tracking process 1100 that may be used bythe object tracking module 206. A first object is captured in a firstimage and detected (step 1102). A first signature of the first object isproduced (step 1104). The first signature may include one or more of thefeatures described above. The first object may be included in the listof tracked objects. A second object is captured in a second image anddetected (step 1106). The object tracking module 206 may use the motionmodeling module 1102 to rank the second object's likelihood ofcorresponding to the first object based upon the first object'sestimated trajectory and velocity (step 1108). A second signature of thesecond object is produced (step 1110). The second signature may includeone or more of the features described above. Step 1110 need not occurafter step 1108 (i.e., the second signature may be produced before themotion modeling module 1102 is used). The first and second signaturesare combined as described above (step 1112). For example, the differentfeatures (e.g., histograms) are combined. The combined signatures aretransmitted to the match classifier 218, and the match classifier 218determines whether the first and second signatures match (step 1114).The match classifier 218 declares a match or a non-match, together witha match confidence level. If the match classifier 218 determines thatthe first and second signatures match, the first signature is updated asdescribed below and other metadata may be generated such as, but notlimited to, motion information (e.g., velocity, trajectory), informationcorresponding to the number of frames in which the first object isdetected, the object's location and size (e.g., the size of a boundingbox surrounding the object), the object's type (e.g., classificationlabel determined by the temporal object classification module 208), aclassification confidence level, and a match confidence level.

As described above, the match classifier 218 may use object signaturesto determine whether objects match (i.e., whether objects are instancesof each other). Different signatures may be generated for each object.For example, a single-frame signature and a cumulative signature may beused by the match classifier 218 during tracking. A single-framesignature (e.g., an instantaneous signature) may represent an object'sappearance in a particular video frame (i.e., a single image of theobject). A single-frame signature may include one or more of thefeatures described above (e.g., histograms) computed based upon thecorresponding frame. A single-frame signature may be generated for everyinstance of an object. The second signature described in the previousparagraph may be a single-frame signature. A cumulative signature may bea weighted average of a sequence of single-frame signatures. Acumulative signature may represent the appearance of an object beingtracked (e.g., an object in the list of tracked objects). Using acumulative signature for a tracked object rather than a single-framesignature may be more robust to noise because a cumulative signature isaveraged over multiple frames. The first signature described in theprevious paragraph may be a cumulative signature.

A cumulative signature may be updated whenever a tracked object ismatched to a new instance by the match classifier 218. For example, thecumulative signature may be updated in real-time as new images of anobject are captured. Moreover, the contribution of each single-framesignature may be reduced (e.g., exponentially reduced) corresponding totime. Thus, the most recent single-frame signatures may contribute themost to a cumulative signature. A cumulative signature may berepresented mathematically as follows:c _(object) =ρc _(object)+(1−ρ)i _(observation)  (13)where c_(object) represents the cumulative signature, i_(observation)represents a single-frame signature corresponding to an instance of anobject, and ρ is a discount factor in which 0<ρ<1. When an object isfirst detected, the cumulative signature may be equal to thesingle-frame signature, and may thereafter correspond to equation (13)for subsequent detections.

A third type of signature—an average signature—may also be generated foran object. Like a cumulative signature, an average signature of anobject includes contributions from single-frame signatures of theobject. Unlike the cumulative signature, however, the single-framesignatures contributing to the average signature are not weighteddifferently. In other words, the contributions from single framesignatures are not reduced according to time. Thus, an average signaturemay represent the sum of a set of single-frame signatures divided by thenumber of single-frame signatures of the set. The following equation isa mathematical representation of an average signature a_(object) of anobject:

$\begin{matrix}{a_{object} = {\frac{1}{n}{\sum\limits_{1}^{n}k_{i}}}} & (14)\end{matrix}$where the object has been tracked for n frames and k_(i) represents thesingle-frame signature for the i^(th) frame. The average signature maybe updated in real-time as new single-frame signatures are generated.

The category or class of objects may also be taken into considerationduring tracking. In many applications, specific categories or classes(e.g., humans, vehicles, animals) of objects are tracked. The objectcategorization module 216 is operable to categorize an object in eachframe as either one of the classes of interest or as “unknown.” Theobject categorization module 216 is operable to use object classinformation and other information to generate an order in which trackedobjects are matched with objects of a current frame. For example, the“importance” of a tracked object may correspond to the number of framesin which the tracked object was classified as something other than“unknown.” Thus, noisy observations corresponding to moving leaves,water, camera noise and the like may have low importance because thenoisy observations may usually be classified as “unknown.” The objectcategorization module 216 may attempt to match noisy observations last,which may reduce the possibility of erroneously matching an “unknown”observation to a object of interest, such as a human or vehicle. Theobject categorization module 216 may be useful when a large number ofobjects are captured in a current frame and sufficient resources may notbe available to track all of the objects. In that case, the “importance”of tracked objects may be used to determine an order for tracking inwhich the most important objects are tracked first.

FIG. 12 is a hybrid block diagram and flow chart of a method 1200 oftracking an object using class information and the modules involved inthat method. Objects are detected in a current frame and the objectclassification module 210 classifies the objects of the current frame aseither a member of one of the object classes (e.g., human, vehicle) oras “unknown.” Data representing the classified objects 1202 are suppliedto the motion modeling module 902. Data representing a list of trackedobjects are also supplied from the object categorization module 216 tothe motion modeling module 902. The tracked objects are objects thathave been observed in recent frames. The object categorization module216 identifies the “importance” of each tracked object based upon thenumber of frames in which the temporal object classification module 208indicates that the tracked object was classified as a member of a classof interest. For example, a user may specify that humans and/or vehiclesare classes of interest. Thus, the more frames in which a tracked objectis classified by the object classification module 210 as a human orvehicle, the greater the “importance” of the tracked object. The objectcategorization module 216 may sort the list of tracked objects accordingto importance. The data supplied from the object categorization module216 to the motion modeling module 902 indicates the importance of eachof the tracked objects and also includes motion information (e.g.,position, trajectory, velocity) for each of the tracked objects.

The motion modeling module 902 receives the data representing the listof tracked objects and the data representing the classified objects1202. The motion modeling module 902 begins with the most importanttracked object from the list (e.g., a first tracked object) and ranksthe classified objects 1202 based on their likelihood of correspondingto the first tracked object. Signatures of likely matches are combinedwith the signature of the first tracked object, and the combinedsignatures are transmitted to the match classifier 218 to determinewhether a classified object 1202 matches the first tracked object. If aclassified object 1202 matches the first tracked object, motioninformation (e.g., position, trajectory, velocity) and appearanceinformation (e.g., cumulative signature) of the first tracked object areupdated (step 1204). Additionally, when a classified object 1202 matchesthe first tracked object, the class of the classified object istransmitted to the temporal object classification module 208 so that the“importance” of the first tracked object can be updated. If none of theclassified objects 1202 match the first tracked object, information isgenerated to indicate a non-match. The information indicating anon-match may be transmitted to the temporal object classificationmodule 208 and the “importance” of the first tracked object may beupdated.

If a tracked object is not matched for a predetermined amount of time(e.g., the object may have left the scene), the list of tracked objectsmay be updated by removing the tracked object (step 1206). When atracked object is removed from the list, the object's information (e.g.,identification number, signature, class) may be transmitted to themetadata database 112 for storage. Additionally, if a classified object1202 does not match any of the tracked objects, the classified object1202 may be added to the list of tracked objects (step 1206).

The camera system 100 may also be operable to perform inter-cameratracking, or tracking between multiple image capturing devices 102.FIGS. 13A and 13B are hybrid block diagrams and flow charts ofinter-camera tracking processes 1300 and 1301 used by, respectively, atransmitting camera and a receiving camera. A first image capturingdevice 102 (e.g., the transmitting camera) detects an object in itsfield of view and tracks the object for a number of frames. The trackedobject 1302 may have its signature routed to other image capturingdevices 102 (e.g., receiving cameras) (step 1304). The signature that isrouted may be an average signature of the tracked object 1302. Thesignature may be transmitted to all image capturing devices 102 of thecamera system 100, or to certain image capturing devices 102 accordingto a routing table 1306. The routing table 1306 may include informationcorresponding to the probability that an object has appeared or willappear in a receiving camera's field of view within a certain timeperiod after or while the object appears in the transmitting camera'sfield of view. The routing information is generated over time as thetransmitting camera transmits signatures to receiving cameras, and asthe receiving cameras report back to the transmitting camera matchnotification information 1308 indicative of matches and/or non-matches.For example, a receiving camera located far away from the transmittingcamera may rarely, if ever, report a match because an object beingtracked by the transmitting camera may not be captured by the receivingcamera within a predetermined time period (e.g., the object cannot movefast enough between the transmitting camera's field of view and thereceiving camera's field of view). Thus, probability information may beused to selectively transmit signatures to certain cameras so as to notburden other cameras with matching tasks when the probability of theother cameras capturing the object within a predetermined time period islow.

The match notification information 1308 is received by a matchnotification responder 1310 that uses the match notification information1308 to update the probabilities of the routing tables. For example, ifa receiving camera reports a match, the match notification responder1310 may increase the probability in the routing table corresponding tothat receiving camera. The match notification information 1308 may alsoinclude signature information corresponding to the object matched by areceiving camera. The match notification responder may update thesignature of the object when a receiving camera detects a match andstore the updated signature in the metadata database 112. Informationidentifying the receiving camera that matched the object may also bestored in the metadata database 112.

As shown in FIG. 13B, an object signature transmitted from thetransmitting camera to a second image capturing device 102 (e.g.,receiving camera) is stored in a match database 1312 of the receivingcamera. The match database 1312 may be part of the storage system 100.The signature of the object is combined with one or more activesignatures of objects being tracked by the receiving camera. The activesignatures of the objects being tracked may be average signatures andmay be stored in the metadata database 112 of the receiving camera. Thecombined signatures are transmitted to the match classifier 218 of thereceiving camera, and the match classifier 218 attempts to match thesignatures for a predetermined number of frames (step 1314). If thematch classifier 218 determines that the transmitted signature does notmatch one of the active signatures for the predetermined number offrames, the transmitted signature is removed from the match database1312 (step 1316). The routing table of the receiving camera is updated(e.g., the probability of a match corresponding to the transmittingcamera is decreased), and the receiving camera notifies the transmittingcamera of the non-match (step 1318). If the match classifier 218determines that the transmitted signature matches one of the activesignatures, the corresponding active signature may be updated (e.g., thetransmitted signature is averaged with the active signature) and storedin the receiving camera's metadata database 112, and/or the transmittedsignature may be stored in the receiving camera's metadata database 112as a new signature for the object (step 1320). The routing table of thereceiving camera is then updated (e.g., the probability of a matchcorresponding to the transmitting camera is increased) and the receivingcamera notifies the transmitting camera of the match (step 1318).

Inter-camera tracking may be used for automatic camera calibration. Forexample, if the location of an object in the field of view of each ofmultiple image capturing devices 102 is known, and if the approximatesize of the object is known, then the distance between the imagecapturing devices 102 and their relative orientation can be calculated.Following calibration, the location of an object simultaneously trackedby the multiple image capturing devices 102 can be determined inthree-dimensional (3-D) “world” coordinates. Thus, the system cananalyze motion of objects not only in a two-dimensional image plane, butalso in 3-D.

By implementing the object tracking module 206 and the match classifier218, object tracking may be significantly improved over known trackingsystems. Moreover, data generated by the object tracking module 206 maybe supplied to the object detection module 204 and the objectclassification module 210 to improve object detection and objectclassification. Additionally, objects may be accurately tracked inreal-time.

Indexing

The signatures of objects act as index elements for the images or videoclips in which the objects are captured. The index elements enableretrieval of the video clips of the objects when the objects areselected. The signatures may be stored in the metadata database 112. Thesignatures may be compact compared to the images or video clipsassociated with the objects. Moreover, the signatures may enableobject-based searching. Once a signature is generated for an object,tracking and searching processes may be independent of video images ofthe object. Thus, video information need not be uncompressed to performsearches, and the search process may be resilient to uncompressedartifacts and/or poor video quality.

Each object may have more than one signature associated with it. Forexample, an object may have multiple single-frame signatures andcumulative and average signatures comprising the single-framesignatures. An object's average signature may be useful for acting as anindex element for the object.

A single average signature computed over the entire duration in which anobject is tracked, however, may not be suitable to enable object-basedsearching because the appearance of an object may be multi-modal or mayvary over time (e.g., a front view of an object versus a side view,lighting changes, views of different portions of the object). Thus, theobject indexing module 212 is operable to determine when to generate anew average signature for an object. Moreover, the object indexingmodule 212 may be operable to call on the match classifier 218 todetermine whether an object's appearance has changed enough such that anew average signature should be generated for the object.

FIG. 14 is a block diagram showing how the match classifier 218 may beused for indexing. When a new single-frame signature 1402 of an objectis generated from a new instance of an object and the new instance ismatched by the match classifier 218 with a tracked object, the trackedobject's cumulative signature 1404 is updated with respect to thesingle-frame signature 1402. After the cumulative signature 1404 isupdated, the cumulative signature 1404 is combined with the trackedobject's average signature 1406. The combined signatures 1408 issupplied to the match classifier 218. If the match classifier 218determines that the cumulative signature 1404 and the average signature1406 match, the average signature is updated with respect to thesingle-frame signature 1402 (block 1410). If the match classifier 218determines that the cumulative signature 1404 and the average signature1406 do not match, a new average signature is created using thesingle-frame signature 1402 (block 1412). The new average signature maybe stored in the metadata database 112 as a second index element for theobject. If the tracked object has multiple average signatures, thecumulative signature 1404 may be compared to each of the multipleaverage signatures, and a new average signature may be created when thecumulative signature 1404 does not match any of the multiple averagesignatures. As described above, the same match classifier 218 used totrack objects may be used to detect appreciable changes in an object'sappearance, and, thus, a new index element may be generated by using thematch classifier 218.

Searching

The signatures or index elements stored in the metadata database 112 mayfacilitate searching a large database of objects quickly for a specificobject because actual pixel information from video images does not needto be reprocessed. The object search module 214 may use the same matchclassifier 218 used for tracking and indexing to search for a specificobject. The match classifier 218, together with the signatures ofobjects, enable object-based searches in both historical video andreal-time video feeds.

FIG. 15 is a pictorial diagram of a search process 1500 that may beperformed by the camera system 100 by way of example only; the process1500 is operable with any suitable camera system. One of the imagecapturing devices 102 captures an image of a reference object, and theimage of the reference object is presented on the display 114 of theuser interface 104. The reference object may be highlighted for the userby, for example, presenting a box outline around the image of thereference object. The user may use the input device 116 to select thereference object. For example, the input device 116 may include a mouse,and the user may click a pointer on the image of the reference objectusing the mouse. Thus, the image of the reference object may beconsidered a hyper-tag (somewhat similar to a hyper-link in an HTMLpage). When the user selects the reference object, the selection ismapped to the reference object's identification number generated by theobject tracking module 206 of the image capturing device 102 thatcaptured the image of the reference object (step 1502). The referenceobject's identification number is used by the object search module 214to retrieve the reference object's signature from the metadata database112 (step 1504). The signature retrieved may be the reference object'saverage signature, and if the reference object has multiple averagesignatures, the multiple average signatures may be retrieved.

In addition to the average signature, one or more fast signatures of thereference object may be retrieved from the metadata database 112. A fastsignature may be a simple appearance descriptor computed from an averagesignature. The fast signature may be computed by discarding some of thefeatures contained in the average signature by reducing the number ofbins in a histogram, for example, or by computing statistics over thefeatures (e.g., maximum or entropy statistics). The fast signature maybe used to quickly reduce the search space of stored signatures byremoving candidate objects that most likely do not match the firstobject. In addition to the fast signature, other criteria may be used bythe object search module 214 to quickly prune out obvious non-matches.For example, the object search module 214 may use the reference object'sclass to eliminate candidate objects. Moreover, the object search module214 may eliminate candidate objects based on time (e.g., eliminatecandidate objects that were detected prior to a predetermined timeframe) and/or based on one or more rule violations associated with thereference object (e.g., eliminate candidate objects that did not triggersimilar rule violations).

The fast signature(s) of the first object are transmitted to the otherimage capturing devices 102 over the camera network 108. The metadatadatabases 112 of the other image capturing devices 102 are searched forfast signatures of candidate objects that match the fast signature(s) ofthe reference object. The object search modules 214 of the other imagecapturing devices 102 compare the fast signature(s) of the referenceobject with the fast signatures of the candidate objects by computing adistance measure between the signatures. The distance measure may thenbe compared to a predetermined threshold. The threshold may be set suchthat the false negative rate (e.g., rate of discarding matches) is low,but the false positive rate (e.g., rate of accepting non-matches) may behigh. If the object search modules 214 determine that one or more fastsignatures of the candidate objects match the fast signature of thereference object, the object search modules 214 retrieve the averagesignatures of the corresponding candidate objects.

The average signatures of the candidate objects are transmitted to theimage capturing device 102 that captured the image of the referenceobject or to the remote storage/processing unit 106 (step 1510). Thematch classifier 218 of the image capturing device 102 or the matchclassifier 218 of the remote storage/processing unit 106 is used tocompare the average signature(s) of the reference object to the averagesignatures of the candidate objects (step 1512). The match classifier218 compares the average signatures to discard non-matches and to rankmatches based on the match confidence level associated with the matches.Once the matches are ranked, a list including representations of theranked matches may be presented on the display 114 (step 1514).Additionally, video images corresponding to the candidate objects of thematches are retrieved over the camera network 108 from one or more ofthe storage systems 110, or from the remote storage/processing unit 106.The user may select a representation of a candidate object from thelist, and the video images of the selected candidate object may bepresented on the display 114.

The metadata database 112 of the image capturing device 102 thatcaptured the image of the reference object may also be searched toidentify signatures of potential matches, similar to the process 1500described above. For example, the object search module 214 of the imagecapturing device 102 may search the fast signatures first and then callon the match classifier 218 to compare the average signature of thereference object to the average signatures of the potential matches.Moreover, although process 1500 includes matching with the matchclassifiers 218 located at either the image capturing device 102 thatcaptured the image of the reference object or the remotestorage/processing unit 106, a match classifier 218 associated with anyother image capturing device 102 may be used. In other words, matchingmay be performed at one or more image capturing device 102 or at theremote storage/processing unit 106.

The match classifier 218 and the object search module 214 enable a formof searching that may be referred to as “click and search.” A user maybe reviewing historical video from a variety of image capturing devices102. If a particular object appears interesting, the user can search forall objects that look like the particular object. Moreover, video clipsof look-a-like objects may be played for the user. The search may beperformed over all the image capturing devices 102 of the camera system100 or over any user-specified subset. Because the video does not haveto be reprocessed to perform a search, the search may be performedquickly.

The match classifier 218 and the object search module 214 also enable aform of searching that may be referred to as “click and track.” Inaddition to performing searches with respect to historical video,searches may be performed for instances of an object that has justappeared in a live video feed. Much like intra-camera tracking, the userinterface 104 requests (in response to a user selection) that the imagecapturing device 102 that captured an object of interest transmit thesignature of the object the image capturing devices 102 in the network108. If a match is found, the user interface 104 is notified and theimages of the object of interest are presented on the display 114.

While reviewing video data retrieved from an object search, a user canvalidate or invalidate a match. For every match that is validated by auser, a link in the metadata database 112 may be created and thecorresponding signatures may be updated. For invalid matches, signaturesmay be designated as exclusions to matching against the object'ssignature. User validation data can be stored in the metadata databaseand retrieved later to re-train match classifier 218, resulting infurther improvement of search and inter-camera tracking results as wellas intra-camera tracking results. Additionally, user validation may beused to tune search results

Applications

The systems and methods described above may be used to create a numberof applications. Examples of some of these applications are presented inthe following paragraphs.

Compared to known systems that create tracking information only when arule established by a rules engine has been violated, the camera system100 creates rich tracking metadata whether a rule is violated. Thisenables post searching, or searching video images that were capturedbefore a rule is violated. Moreover, because metadata is generated thatincludes object appearance characteristics, an object may be recognizedas a member of a newly created object class even though an image of theobject was captured prior to creation of the new object class.

Known tracking systems have problems with losing track of objects. Forexample, if a person walks behind a park bench, the person's legs mightnot be visible. A known tracking system might consider the person tooshort to be a person. As a result, the person may be tracked until theperson walks behind the park bench. Because the camera system 100 mayuse many different appearance characteristics, the camera system 100 mayavoid losing track of objects when they are still present within thefield of view. Moreover, the match classifier 218 may make tracking moreadaptable to environmental changes and may improve detection acrossdifferent camera types (and from camera to camera in general) comparedto known systems.

Because the match classifier 218 is trainable, new features of objectsmay be selected to improve detection and tracking accuracy. The matchclassifier 218 may learn the features that are most important in thepre-deployment training phase, and may also learn while in the field toimprove tracking accuracy.

By establishing semantic links between video streams and objectsdetected, a video history can be created for a particular object. Forinstance, by selecting a human object, a user may automatically summonvideo clips showing where the person had been detected previously byother cameras. The user may then notice companions of the person inquestion, and may select those companions and view their videohistories. Because metadata corresponding to the object's appearancesignature is linked in the database with video data corresponding to thelocation where it was detected, the image itself may be used as aselectable link for searching the database.

When an object is captured by one image capturing device 102, thesystems and methods described may also be used to notify and engage allother image capturing devices 102, so that they may automatically beginsearching for the same object and create an alert if it is detected. Forexample, an individual person may be followed through a casino monitoredby dozens of cameras with adjacent, overlapping fields of view, by justclicking on an image and instructing the system to track the imageacross all cameras. As soon as an appearance signature of the person isdetected, the system automatically directs live video data from thecorresponding camera to a monitor that allows security personnel tovisually track the person. As the person moves into the field of view ofthe next camera, the video feed is automatically switched so that it isnot necessary for security personnel to switch back and forth betweencameras to continue tracking the person's path.

The camera system 100 may automatically recognize pedestrians leaving aparked car, and can compare them later to people entering the car. If aperson entering a car is not from the original group who arrived in thecar, security personnel may be alerted. Video clips are automaticallysent with the alert, so that it is easy to review and quickly determinewhether there is a problem. A security guard may then opt to eitherinform the car owner (if the car is registered by license plate number)or summon police.

With the ability to track objects, fraud can be detected by followingthe path of individual shoppers through a retail store. With thiscapability the system disclosed may be taught to verify whether shoppersexiting a store with merchandise have visited a checkout counter andpurchased the merchandise before leaving the store. The system may alsodetermine whether customers at a return counter came straight from theircars, or whether they first passed through the store, selected productsfrom the store, and attempted to return them as though they hadpurchased the products. The return counter attendant can be shown a clipof the customer entering the store and their path through the store, asverification.

When combined with object matching, performance of facial recognitionand vehicle license plate recognition systems can be improved. Objectmatching may provide good tracking capability by overall appearance andmovement and can track people in any orientation, not just when they arealigned to face the camera.

The camera system 100 may also be configured to automatically recognizeabnormal activity by monitoring vehicle and pedestrian patterns overdays or weeks. High risk facilities concerned about potential terroristactivities may benefit from this feature. If the camera system 100repeatedly recognizes a vehicle parked for an extended period of timewith no one exiting the vehicle, an alert of suspicious activity may begenerated to summon security personnel to review the video data.Alternatively, object matching combined with a facial recognition systemmay recognize individuals who walk around a facility repeatedly. Suchaction may catch a criminal casing a facility. In another example, ifuniformed employees regularly visit a special lunch room or inventoryroom, for example, the camera system 100 can recognize this pattern.Later, if someone not wearing such a uniform enters one of these rooms,the camera system 100 may respond with an automatic alert.

The camera system 100 may be used to measure how many people arestanding in a line at, for example, a cash register and to generate analert corresponding to a rule of a rules engine. Additionally, thecamera system 100 may be used to measure how long each person has beenstanding in the line because the system 100 can track each individual.If a person has been standing in the line too long, a rule of the rulesengine may be violated and an alert may be generated to summonassistance (e.g., an alert to open a new cash register line). Moreover,statistics may be generated for managers to help measure customerservice performance.

The camera system 100 may be used to help prevent child abductions atretail businesses for example. The camera system 100 may recognizechildren when they enter a store and an adult accompanying the child.The camera system 100 may also recognize an adult the child stays closeto while walking through the store. If the child leaves the storewithout the adult accompanying the child, an alert may be created. Thecamera system 100 may also identify the location of the adult so thatshe/he can be notified.

The camera system 100 may be used to locate lost children at retailbusinesses. For example, if a parent loses its child and notifies asecurity guard, the security guard can find an image of the parent,click on the image to search through other video images and identify animage when the parent entered the store with the child. The securityguard may then click on an image of the child and search the imagecapturing device 102 that most recently captured an image of the child.This quickly locates the child's location in the store.

The camera system 100 may be use to prevent a problem called “sweetheartfraud,” which happens when a checkout person intentionally swipes thebar code of a low cost item while acting as though they are swiping thebar code of a more expensive item. The camera system 100 may beconnected to the point-of-sale (POS) data outputs of a cash register andmay compare the appearance of products with the bar code information.Over a period of time, the camera system 100 may automatically create inthe metadata database 112 appearance signatures for different products.After the camera system 100 has reached a level of accuracy for productrecognition, which the camera system 100 can determine itself, it canautomatically begin creating alerts whenever the bar code informationdoes not match the image being scanned.

The camera system 100 may automatically tag video for higher prioritybased on its similarity to footage in which a critical event isdetected. For example, if a laptop is stolen from a work area, thecamera system 100 may be able to automatically recognize the laptop fromhistorical video and may recognize the person closest to the laptop whenit went missing. The camera system 100 may then automatically designateall video images of the suspicious person to be of higher priority forstorage, retention, or retrieval and review by security personnel.

CONCLUSION

The methods and systems illustrated and described herein can exist in avariety of forms both active and inactive. For example, they can existpartly or wholly as one or more software programs comprised of programinstructions in source code, object code, executable code or otherformats. Any of the above can be embodied in compressed or uncompressedform on a computer-readable medium, which include storage devices.Exemplary computer-readable storage devices include conventionalcomputer system RAM (random access memory), ROM (read only memory),EPROM (erasable, programmable ROM), EEPROM (electrically erasable,programmable ROM), flash memory and magnetic or optical disks or tapes.

The terms and descriptions used above are set forth by way ofillustration only and are not meant as limitations. For example, thematch classifier 218 may be part of—and the classification methods maybe performed at—a remote processing unit, such as the remotestorage/processing unit 106 (FIG. 1), a computer associated with theuser interface 104, another node in the camera network 108, or anotherserver, such as one at a central location or on another network. Thoseskilled in the art will recognize that many variations, enhancements andmodifications of the concepts described herein are possible withoutdeparting from the underlying principles of the invention. The scope ofthe invention should therefore be determined only by the followingclaims and their equivalents.

1. A camera system comprising: an object detection module receivingimage data produced by an image capturing device, the image datarepresenting multiple images of a field of view of the image capturingdevice, the object detection module being operable to detect objectsappearing in one or more of the multiple images; an object trackingmodule connected to the object detection module and operable totemporally associate instances of a first object detected in a firstgroup of the multiple images, the first object having a first signaturerepresenting features of the first object derived from the images of thefirst group; and a match classifier connected to the object trackingmodule for matching object instances and including a set of matching andnon-matching parameters populating a feature classification space, thematch classifier being operable to match object instances by analyzingdata derived from the first signature of the first object and a secondsignature of a second object, the second object being detected in asecond image distinct from the images of the first group, and the secondsignature representing features of the second object derived from thesecond image, the match classifier analyzing with reference to the setof match and non-match parameters data derived from combinations of thefeatures represented by the first and second signatures to determinewhether the second signature matches the first signature and therebyindicate whether the first and second objects correspond to the samephysical object.
 2. The camera system of claim 1, wherein the secondobject is an instance of the first object.
 3. The camera system of claim1, wherein the second signature is generated in real-time.
 4. The camerasystem of claim 1, wherein the match classifier includes N decisionsteps configured in a cascade configuration, the N decisions stepsoperable to determine whether the first object and the second objectmatch, wherein at least one of the N decision steps is operable toeither (a) accept the first object and the second object as a match, (b)reject the first object and the second object as a match, or (c) call ona next step to determine whether the second object matches the firstobject.
 5. The camera system of claim 4, wherein said at least one ofthe N decision steps includes a stage for mapping a portion of the datato a scalar value that correlates to a match confidence.
 6. The camerasystem of claim 5, wherein the first signature represents appearancefeatures of the first object and includes a first edge orientationhistogram of the first object, and the second signature representsappearance features of the second object and includes a second edgeorientation histogram of the second object.
 7. The camera system ofclaim 6, wherein the first and second edge orientation histograms areproduced using a steerable filter.
 8. The camera system of claim 6,wherein the first and second edge orientation histograms are combined toproduce the portion of the data.
 9. The camera system of claim 8,wherein the first and second edge orientation histograms are combined byconcatenation.
 10. The camera system of claim 6, wherein the firstsignature includes a first group of multiple edge orientation histogramscorresponding to a first set of different spatial regions of the firstobject, and the second signature includes a second group of multipleedge orientation histograms corresponding to a second set of differentspatial regions of the second object, the different spatial regions ofthe second set corresponding to the different spatial regions of thefirst set.
 11. The camera system of claim 6, wherein the first signatureincludes a first hue histogram and a first intensity histogram of thefirst object, and the second signature includes a second hue histogramand a second intensity histogram of the second object.
 12. The camerasystem of claim 6, wherein the first signature includes a first group ofmultiple hue histograms corresponding to a first set of differentspatial regions of the first object, and the second signature includes asecond group of multiple hue histograms corresponding to a second set ofdifferent spatial regions of the second object, the different spatialregions of the second set corresponding to the different spatial regionsof the first set.
 13. The camera system of claim 6, wherein the firstsignature includes a first group of multiple intensity histogramscorresponding to a first set of different spatial regions of the firstobject, and the second signature includes a second group of multipleintensity histograms corresponding to a second set of different spatialregions of the second object, the different spatial regions of thesecond set corresponding to the different spatial regions of the firstset.
 14. The camera system of claim 1, wherein the match classifierincludes N decision steps configured in a cascade configuration, each ofthe N decision steps including one or more stages for mapping portionsof the data to scalar values, wherein a first one of the stages includesa first discriminant function for determining a first scalar value, asecond one of the stages includes a second discriminant function fordetermining a second scalar value, and the first and second discriminantfunctions are selected from a group consisting of plural discriminantfunctions of different types.
 15. The camera system of claim 1, whereinthe first signature is a cumulative signature of the first objectderived from the images of the first group, and the second signature issingle-frame signature of the second object derived from the secondimage.
 16. The camera system of claim 15, wherein the cumulativesignature of the first object is a weighted average of single-framesignatures of the first object derived from the images of the firstgroup.
 17. The camera system of claim 1, wherein the object trackingmodule includes a motion modeling module operable to estimate changes oflocation of objects between sequential images of the field of view,wherein the motion modeling module is operable to be used in conjunctionwith the match classifier to determine whether the second object is aninstance of the first object.
 18. The camera system of claim 1, furthercomprising multiple image capturing devices connected to one another ina network, the multiple image capturing devices including first andsecond image capturing devices, the first group of images being capturedby the first image capturing device, and the second image being capturedby the second image capturing device.
 19. The camera system of claim 18,wherein the first image capturing device includes a housing, the matchclassifier is contained within the housing, and the second imagecapturing device is operable to transmit to the match classifier of thefirst image capturing device the second signature of the second object.20. The camera system of claim 19, wherein the match classifier isoperable to determine that the second signature transmitted from thesecond image capturing device and the first signature of the firstobject match, and the first image capturing device is operable to notifythe second image capturing device of the match.
 21. The camera system ofclaim 18, wherein the match classifier is part of a processor that isremote from the first and second image capturing devices, and the firstand second image capturing devices are operable to transmit,respectively, the first and second signatures to the processor.
 22. Thecamera system of claim 21, wherein the processor is operable to transmita notification to one or both of the first and second image capturingdevices indicating that the first and second signatures match.
 23. Thecamera system of claim 1, wherein the second object is one of multipleobjects detected in the second image, and the first object is one ofmultiple tracked objects being tracked by the object tracking module,the camera system further comprising: an object classification module,the object classification module being operable to classify objects asmembers or non-members of an object class; and an object categorizationmodule connected to the match classifier and the object classificationmodule, the object categorization module being operable to determine anorder in which the match classifier attempts to match the trackedobjects with the multiple objects detected in the second image, theorder being based on classifications of the tracked objects determinedby the object classification module.
 24. The camera system of claim 1,wherein the first signature is a cumulative signature representing aweighted average of single-frame signatures of the first object derivedfrom the images of the first group, the second signature represents asingle-frame signature of the second object derived from the secondimage, and the match classifier determines that the second object is aninstance of the first object.
 25. The camera system of claim 24, furthercomprising: a database for storing average signatures of detectedobjects, the average signatures being index elements of the detectedobjects; and an object indexing module connected to the match classifierand the database, the object indexing module being operable to generatea first average signature for the first object, the first averagesignature representing an average of the single-frame signatures of thefirst object, wherein the object indexing module is operable to supplythe first average signature to the database for storage.
 26. The camerasystem of claim 25, wherein said data derived from the first signatureof the first object and the second signature of the second object is afirst set of data, the object indexing module is operable to call on thematch classifier to determine whether the second signature of the secondobject is to contribute to the first average signature, and the matchclassifier is operable to analyze a second set of data representing thefirst signature and the first average signature to determine whether thefirst signature and the first average signature match.
 27. The camerasystem of claim 26, wherein the object indexing module is operable toupdate the first average signature with contributions from the secondsignature when the match classifier determines that the first signaturematches the first average signature.
 28. The camera system of claim 26,wherein the object indexing module is operable to generate a secondaverage signature for the first object when the match classifierdetermines that the first signature does not match the first averagesignature, the second average signature includes contributions from thesecond signature, and the object indexing module is operable to supplythe first and second average signatures to the database for storage. 29.The camera system of claim 28, wherein the first average signaturerepresents a first view of the first object, and the second averagesignature represents a second view of the first object.
 30. The camerasystem of claim 1, further comprising: a database for storing signaturesof detected objects, the database providing an index to images of thedetected objects; a user interface including a display for presentingthe multiple images of the field of view to a user, the user interfaceincluding an input device enabling user selection of an object appearingin one or more of the multiple images, wherein an image of the firstgroup including a representation of the first object is presented on thedisplay, and the first object is selected by the user; and a searchmodule connected to the user interface, the database, and the matchclassifier, the search module being operable to search the signaturesstored in the database to identify whether the first object appears in asecond group of images of the detected objects, the search module beingoperable to call on the match classifier to determine whether the firstsignature of the first object matches one or more of the signaturesstored in the database.
 31. The camera system of claim 30, wherein thematch classifier is operable to generate confidence levels of matchesmade between the first signature and the signatures stored in thedatabase, and the search module is operable to generate a list ofmatches organized according to confidence levels.
 32. The camera systemof claim 31, wherein the signatures stored in the database includeaverage signatures of the detected objects representing averages ofsingle-frame signatures derived from images of the second group in whichthe detected objects appear, and the signatures stored in the databaseinclude fast signatures of the detected objects representing simplifiedversions of the average signatures.
 33. The camera system of claim 32,wherein the first object includes a fast signature representing asimplified version of the first signature, and the search module isoperable to compare the fast signature of the first object with the fastsignatures stored in the database to identify a group of the detectedobjects that do not correspond to the first object.
 34. The camerasystem of claim 33, wherein said data derived from the first signatureof the first object and the second signature of the second object is afirst set of data, and the match classifier is operable to analyze asecond set of data representing the first signature of the first objectand the average signatures of the detected objects that are not membersof the group that do not correspond to the first object.
 35. The camerasystem of claim 31, wherein when the match classifier determines thatthe first signature matches a signature of a first one of the detectedobjects, the search module is operable to retrieve image datarepresenting an image of said first one of the detected objects fortransmission to the user interface.
 36. The camera system of claim 31,further comprising multiple image capturing devices connected to oneanother in a network, the multiple image capturing devices includingfirst and second image capturing devices.
 37. The camera system of claim36, wherein the images of the first group are captured by the firstimage capturing device, and the images of the second group are capturedby the second image capturing device.
 38. The camera system of claim 37,wherein the second image capturing device includes a housing, and thematch classifier is contained within the housing.
 39. The camera systemof claim 37, wherein the match classifier is part of a processor remotefrom the first and second image capturing devices.
 40. The camera systemof claim 1, further comprising: a user interface including a display fordisplaying objects detected by the object detection module and an inputdevice operable to select objects detected by the object detectionmodule in response to user commands; and a search module connected tothe match classifier and the user interface, the search module beingoperable to retrieve stored images of objects selected by the inputdevice, and the search module being operable to call on the matchclassifier to locate a stored image of a selected object.
 41. The camerasystem of claim 40, further comprising an object indexing moduleconnected to the match classifier and the object detection module, theobject indexing module being operable to generate one or more indexelements for objects detected by the object detection module, the objectindexing module being operable to call on the match classifier todetermine whether to generate a new index element for a detected object.42. The camera system of claim 1, wherein the manner in which the matchclassifier operates to match object instances has been determined atleast in part by a training process that automatically configures thematch classifier using a set of possible object features.